fix(sglang): force reasoning when the template prefills the think tag - #11790
fix(sglang): force reasoning when the template prefills the think tag#11790pos-ei-don wants to merge 1 commit into
Conversation
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
localai-org-maint-bot
left a comment
There was a problem hiding this comment.
Good to merge from my review. The prefilled-think-tag detection is scoped correctly, grammar-constrained output is preserved as content, and both parser paths share the same construction logic. Regression coverage, Python syntax checks, and diff checks look clean. @mudler
|
Review pass. The reasoning is correct and matches how sglang's own server handles this via Why it is right: when Qwen3's template appends The subtle part is also right: the Three small things, none blocking:
That last one matters more than it looks, because Conflicts with #11786 on |
Description
Qwen3-style chat templates append the opening
<think>tag to the prompt when thinkingis enabled. The model therefore never generates it and emits only the reasoning text plus the
closing
</think>.sglang's
ReasoningParserkeys off the opening tag:With such a template the entire completion — reasoning, the raw
</think>, and the answer —comes back as
content, andreasoning_contentstays empty no matter howreasoning_parseris configured:
sglang's own OpenAI server solves this with
This backend has no template manager, so the same signal is derived from the rendered prompt:
if it ends with the detector's
think_start_token, the tag was prefilled and the parser isconstructed with
force_reasoning=True.Notes for Reviewers
Why this is conditional rather than always on. Forcing unconditionally breaks the
thinking-off case — a completion with no tags at all is then filed entirely as reasoning and
the answer disappears:
force_reasoningis only passed when it is meant to beTrue, so detector defaults(DeepSeek-R1 already defaults to
True) are untouched.Structured decoding is the exception, and it matters. A grammar applies from the first
token, so the model cannot emit the closing tag even though the template opened the block.
The whole completion is schema output and belongs in
content; forcing there files it asreasoning and returns an empty answer. Measured against a JSON-schema code audit:
10107 characters of "reasoning", zero content. sglang's own server keeps the two apart
for the same reason — its grammar backend owns the reasoning prefix when a reasoning parser
is configured. Hence the
grammar_constrainedguard, with a regression test for it.The construction is factored into
_new_reasoning_parser()so the streaming andnon-streaming paths, which previously built the parser separately, cannot drift apart.
Tested against Qwen3-style models on an sglang backend;
backend/python/sglang/test.pycoversthe prefilled-tag case, the thinking-off case and the grammar case.
Signed commits